Method and system for providing artificial intelligence analytic (aia) services for performance prediction

ABSTRACT

One embodiment of the present invention predicts a vehicular event relating to machinal performance using information obtained from interior and exterior sensors, vehicle onboard computer (“VOC”), and cloud data. The process of predication is able to activate interior and exterior sensors mounted on a vehicle operated by a driver for obtaining current data relating to external surroundings, interior settings, and internal mechanical conditions of the vehicle. After forwarding the current data to VOC to generate a current vehicle status representing real-time vehicle performance in accordance with the current data, retrieving a historical data associated with the vehicle including mechanical condition is retrieved. In one aspect, a normal condition signal is issued when the current vehicle status does not satisfy with the optimal condition based on the historical data. Alternatively, a race car condition is issued when the current vehicle status meets with the optimal condition.

PRIORITY

This application claims the benefit of priority based upon U.S.Provisional Patent Application having an application Ser. No.62/438,268, filed on Dec. 22, 2016, and having a title of “Method andSystem for Providing Artificial Intelligence (AI) Analytic ServicesUsing Cloud and Embedded Data,” and U.S. non-provisional PatentApplication having an application Ser. No. 15/852,567, filed on Dec. 22,2017 and having a title of “Method and System for Providing ArtificialIntelligence Analytic (AIA) Services for Performance Prediction,” whichare hereby incorporated by reference in its entirety.

FIELD

The exemplary embodiment(s) of the present invention relates to thefield of communication networks. More specifically, the exemplaryembodiment(s) of the present invention relates to providing automationrelating to vehicles using artificial intelligence (“AI”) modules andcloud computing.

BACKGROUND

With rapid integration of motor vehicle with wireless network, AI, andIoT (Internet of Things), the demand of intelligent machine and instantresponse is constantly growing. For example, the cars or vehicles whichbecome smarter can assist drivers to operate the vehicles. To implementthe integration of vehicle and AI, some technical pieces, such as datamanagement, model training, and data collection, need to be improved.The conventional machine learning process, for example, is generally anexploratory process which may involve trying different kinds of models,such as convolutional, RNN (recurrent neural network), attentional, etcetera.

Machine learning or model training typically concerns a wide variety ofhyper-parameters that change the shape of the model and trainingcharacteristics. Model training generally requires intensive computationand data collection. With conventional data collection via IoT, AI,real-time images, videos, and/or machine learning, the size of data(real-time data, cloud data, big data, etc.) is voluminous and becomesdifficult to handle and digest. As such, real-time response via machinelearning model with massive data processing can be challenging. Anotherdrawback associated with large data processing and machine learning formodel improvements is that it is often difficult to translate collecteddata into useful information.

SUMMARY

One embodiment of the present invention discloses an artificialintelligence analytic (“AIA”) process of providing a prediction orreport capable of predicting an event relating to vehicle performanceusing data obtained from interior and exterior sensors, vehicle onboardcomputer (“VOC”), and cloud computing. The process activates theinterior and exterior sensors mounted on a vehicle operated by a driverfor obtaining current data relating to external surroundings, interiorsettings, and internal mechanical conditions of the vehicle. Forexample, after enabling a set of outward facing cameras mounted on thevehicle for recording external surrounding images representing ageographic environment, one or more inward facing cameras mounted in thevehicle is initiated for collecting interior images of the vehicle.Also, a set of internal sensors attached to various mechanicalcomponents is activated for measuring temperatures, functionalities, oraudio sounds associated with mechanical components within the vehicle.The process, in one embodiment, is capable of detecting driver'sresponse time based on a set of identified road conditions andinformation from a controller area network (“CAN”) bus of the vehicle.The real-time data relating to vehicle performance, road condition,traffic congestion, and weather condition is recorded. The current datais forwarded to VOC for generating a current vehicle status representingsubstantially real-time vehicle performance in accordance with thecurrent data. The historical data associated with the vehicle includingmechanical condition is retrieved. Note that the historical data isupdated in response to the current data.

A normal condition signal is issued when the current vehicle status doesnot satisfy with optimal condition based on the historical data. In oneaspect, after uploading the current vehicle status to a vehicleperformance predictor which resides at least partially at a cloud via acommunications network, the big data is obtained from the cloud whereinthe big data represents large car samples having similar attributes asthe vehicle. For example, the big data accumulates information from carswith similar brands, similar mileages, similar years, similar geographiclocation, and similar drivers. The current vehicle status is comparedwith the big data and the historical data to assess whether the vehicleoperates in a normal condition. The “race car ready” condition is issuedwhen the current vehicle status meets with the optimal condition basedon the historical data. In one example, the current vehicle status isforwarded to a subscriber for evaluating driver's driving skill. Thecurrent vehicle status can also be forwarded to a subscriber forassessing normal wearing and tearing. Alternatively, a subscriberschedules a maintenance or repair appointment with the driver based onthe current vehicle status. In one embodiment, a manufacture caninitiate a recall for automobiles similar to the vehicle at leastpartially based on the current vehicle status.

An AIA system capable of providing a vehicle performance predictionrelating to an automobile operated by a driver includes sensors, VOC,and cloud network. In one aspect, a group of interior and exteriorsensors mounted on a vehicle and internal mechanical components areconfigured to collect information relating to external surroundings,interior environment, and components conditions. For example, theinterior and exterior sensors include outward facing cameras mounted ona vehicle collecting external images representing a surroundingenvironment in which the vehicle operates and inward facing camerasmounted inside of the vehicle collecting interior images includingoperator facial expression and operator's attention.

VOC is used to generate a current vehicle status which represents acurrent vehicle condition in accordance with the obtained information.The cloud network, in one embodiment, includes a normal conditionmodule, prediction module, and race car module. The normal conditionmodule determines a normal condition for the vehicle based on thecurrent vehicle status, historical vehicle status, and big data, whereinthe big data represents a large sample having similar attributes as thevehicle. The prediction module is capable of predicting vehicle failurein response to the normal condition. The race car module is able toprovide skills and/or mistakes associated with a race car driveroperating the vehicle based on the current vehicle status, historicalvehicle status, and big data which is a large set of accumulated sampleshaving similar characteristics as the vehicle. It should be noted thatthe cloud network also includes a subscription module for facilitatingand enlisting subscribers.

In an alternative embodiment, an AIA system or process able toaccumulate information relating to machinal performance and drivers inaccordance with information obtained from interior and exterior sensors,VOC, and cloud network is capable of activating interior and exteriorsensors mounted inside and outside of a vehicle operated by a driver forcollecting real-time information relating to external surroundings,interior settings, and internal conditions of the vehicle. Afterforwarding the real-time information to the VOC to generate a currentvehicle status representing substantial real-time vehicle performance inaccordance with the real-time information, the current vehicle status isuploaded to the cloud network and retrieving a historical dataassociated with the vehicle including performance data. Upon generatinga vehicle performance report in response to the historical data and bigdata containing relatively large samples having similar attributes asthe vehicle, a maintenance appointment is scheduled by a repair shopwith the driver based on the vehicle performance report. The vehicleperformance report can also be forwarded to a subscribed automobilemanufacturer for vehicle recalls. Moreover, the vehicle performancereport can also be sent to a subscriber owner of the vehicle indicatingcurrent mechanical condition of the vehicle.

Additional features and benefits of the exemplary embodiment(s) of thepresent invention will become apparent from the detailed description,figures and claims set forth below.

BRIEF DESCRIPTION OF THE DRAWINGS

The exemplary embodiment(s) of the present invention will be understoodmore fully from the detailed description given below and from theaccompanying drawings of various embodiments of the invention, which,however, should not be taken to limit the invention to the specificembodiments, but are for explanation and understanding only.

FIGS. 1A-1B are block diagrams illustrating artificial intelligenceanalytic service (“AIAS”) capable of predicting vehicle mechanicalperformance (“VMP”) using a virtuous cycle in accordance with oneembodiment of the present invention;

FIG. 1C is a block diagram illustrating a process of generating reportsrelating to vehicle as well as driver status using an AIA model via avirtuous cycle in accordance with one embodiment of the presentinvention;

FIG. 1D is a block diagram illustrating an integrated developmentenvironment (“IDE”) configured to host various models for providing AIASvia a virtuous cycle in accordance with one embodiment of the presentinvention;

FIG. 1E is a logic diagram illustrating an AIA system configured toprovide a prediction relating to vehicle mechanical performance (“VMP”)via a virtuous cycle in accordance with one embodiment of the presentinvention;

FIG. 1F is a logic diagram illustrating an AIA system facilitatedconfigured to provide a vehicle failure prediction (“VFP”) via avirtuous cycle in accordance with one embodiment of the presentinvention;

FIG. 1G is a logic diagram illustrating an AIA process facilitated byAIA system configured to provide VMP using big data via a virtuous cyclein accordance with one embodiment of the present invention;

FIGS. 2A-2B are block diagrams illustrating a virtuous cycle capable offacilitating AIAS using IA model in accordance with one embodiment ofthe present invention;

FIG. 3 is a block diagram illustrating a cloud based network usingcrowdsourcing approach to improve IA model(s) for AIAS in accordancewith one embodiment of the present invention;

FIG. 4 is a block diagram illustrating an IA model or AIA system usingthe virtuous cycle in accordance with one embodiment of the presentinvention;

FIG. 5 is a block diagram illustrating an exemplary process ofcorrelating data for AIAS in accordance with one embodiment of thepresent invention;

FIG. 6 is a block diagram illustrating an exemplary process of real-timedata management for AI model for AIAS in accordance with one embodimentof the present invention;

FIG. 7 is a block diagram illustrating a crowd sourced application modelfor AI model for AIAS in accordance with one embodiment of the presentinvention;

FIG. 8 is a block diagram illustrating a method of storing AI relateddata using a geo-spatial objective storage for AIAS in accordance withone embodiment of the present invention;

FIG. 9 is a block diagram illustrating an exemplary approach of analysisengine analyzing collected data for AIAS in accordance with oneembodiment of the present invention;

FIG. 10 is a block diagram illustrating an exemplary containerizedsensor network used for sensing information for AIAS in accordance withone embodiment of the present invention;

FIG. 11 is a block diagram illustrating a processing device VOC, and/orcomputer system which can be installed in a vehicle for facilitating thevirtuous cycle in accordance with one embodiment of the presentinvention; and

FIG. 12 is a flowchart illustrating a process of AIA system for AIAScapable of providing VMP in accordance with one embodiment of thepresent invention.

DETAILED DESCRIPTION

Embodiments of the present invention are described herein with contextof a method and/or apparatus for providing prediction services usingcloud data, embedded data, and machine learning center (“MLC”).

The purpose of the following detailed description is to provide anunderstanding of one or more embodiments of the present invention. Thoseof ordinary skills in the art will realize that the following detaileddescription is illustrative only and is not intended to be in any waylimiting. Other embodiments will readily suggest themselves to suchskilled persons having the benefit of this disclosure and/ordescription.

In the interest of clarity, not all of the routine features of theimplementations described herein are shown and described. It will, ofcourse, be understood that in the development of any such actualimplementation, numerous implementation-specific decisions may be madein order to achieve the developer's specific goals, such as compliancewith application- and business-related constraints, and that thesespecific goals will vary from one implementation to another and from onedeveloper to another. Moreover, it will be understood that such adevelopment effort might be complex and time-consuming, but wouldnevertheless be a routine undertaking of engineering for those ofordinary skills in the art having the benefit of embodiment(s) of thisdisclosure.

Various embodiments of the present invention illustrated in the drawingsmay not be drawn to scale. Rather, the dimensions of the variousfeatures may be expanded or reduced for clarity. In addition, some ofthe drawings may be simplified for clarity. Thus, the drawings may notdepict all of the components of a given apparatus (e.g., device) ormethod. The same reference indicators will be used throughout thedrawings and the following detailed description to refer to the same orlike parts.

In accordance with the embodiment(s) of present invention, thecomponents, process steps, and/or data structures described herein maybe implemented using various types of operating systems, computingplatforms, computer programs, and/or general-purpose machine. Inaddition, those of ordinary skills in the art will recognize thatdevices of less general-purpose nature, such as hardware devices, fieldprogrammable gate arrays (FPGAs), application specific integratedcircuits (ASICs), or the like, may also be used without departing fromthe scope and spirit of the inventive concepts disclosed herein. Where amethod comprising a series of process steps is implemented by a computeror a machine and those process steps can be stored as a series ofinstructions readable by the machine, they may be stored on a tangiblemedium such as a computer memory device (e.g., ROM (Read Only Memory),PROM (Programmable Read Only Memory), EEPROM (Electrically ErasableProgrammable Read Only Memory), FLASH Memory, Jump Drive, and the like),magnetic storage medium (e.g., tape, magnetic disk drive, and the like),optical storage medium (e.g., CD-ROM, DVD-ROM, paper card and papertape, and the like) and other known types of program memory.

The term “system” or “device” is used generically herein to describe anynumber of components, elements, sub-systems, devices, packet switchelements, packet switches, access switches, routers, networks, computerand/or communication devices or mechanisms, or combinations ofcomponents thereof. The term “computer” includes a processor, memory,and buses capable of executing instruction wherein the computer refersto one or a cluster of computers, personal computers, workstations,mainframes, or combinations of computers thereof.

One embodiment of the present invention predicts a vehicular eventrelating to machinal performance using information obtained frominterior and exterior sensors, vehicle onboard computer (“VOC”), andcloud data. The process of predication is able to activate interior andexterior sensors mounted on a vehicle operated by a driver for obtainingcurrent data relating to external surroundings, interior settings, andinternal mechanical conditions of the vehicle. After forwarding thecurrent data to VOC to generate a current vehicle status representingreal-time vehicle performance in accordance with the current data,retrieving a historical data associated with the vehicle includingmechanical condition is retrieved. In one aspect, a normal conditionsignal is issued when the current vehicle status does not satisfy withthe optimal condition based on the historical data. Alternatively, arace car condition is issued when the current vehicle status meets withthe optimal condition.

FIG. 1A is a block diagram 100 illustrating artificial intelligenceanalytic service (“AIAS”) capable of predicting vehicle mechanicalperformance (“VMP”) using a virtuous cycle in accordance with oneembodiment of the present invention. Diagram 100 illustrates a virtuouscycle containing a vehicle 102, cloud based network (“CBN”) 104, andmachine learning center (“MLC”) 106. In one aspect, MCL 106 can belocated remotely or in the cloud. Alternatively, MCL 106 can be a partof CBN 104. It should be noted that the underlying concept of theexemplary embodiment(s) of the present invention would not change if oneor more blocks (circuit or elements) were added to or removed fromdiagram 100.

Vehicle 102, in one example, can be a car, automobile, bus, train,drone, airplane, truck, and the like, and is capable of movinggeographically from point A to point B. To simplify forgoing discussing,the term “vehicle” or “car” is used to refer to car, automobile, bus,train, drone, airplane, truck, motorcycle, and the like. Vehicle 102includes wheels with ABS (anti-lock braking system), auto body, steeringwheel 108, exterior or outward facing cameras 125, interior (or 360°(degree)) or inward facing camera(s) 126, antenna 124, onboardcontroller or VOC 123, and operator (or driver) 109. It should be notedthat outward facing cameras and/or inward facing cameras 125-126 can beinstalled at front, side, top, back, and/or inside of vehicle 102. Inone example, vehicle 102 also includes various sensors which sensesmechanical related data associated with the vehicle, vehicle status,and/or driver actions. For example, the sensors, not shown in FIG. IA,can also collect other relevant information, such as audio, ABS,steering, braking, acceleration, traction control, windshield wipers,GPS (global positioning system), radar, sonar, ultrasound, lidar (LightDetection and Ranging), and the like.

VOC or onboard controller 123 includes CPU (central processing unit),GPU (graphic processing unit), memory, and disk responsible forgathering data from outward facing or exterior cameras 125, inwardfacing or interior cameras 126, audio sensor, ABS, traction control,steering wheel, CAN-bus sensors, and the like. In one aspect, VOC 123executes IA model received from MLC 106, and uses antenna 124 tocommunicate with CBN 104 via a wireless communication network 110. Notethat wireless communication network includes, but not limited to, WIFI,cellular network, Bluetooth network, satellite network, or the like. Afunction of VOC 123 is to gather or capture real-time surroundinginformation as well as exterior information when vehicle 102 is moving.

CBN 104 includes various digital computing systems, such as, but notlimited to, server farm 120, routers/switches 121, cloud administrators119, connected computing devices 116-117, and network elements 118. Afunction of CBN 104 is to provide cloud computing which can be viewed ason-demand Internet based computing service with enormous computing powerand resources. Another function of CBN 104 is to improve or inferredattentional labeled data via correlating captured real-time data withrelevant cloud data.

MLC 106, in one embodiment, provides, refines, trains, and/ordistributes models 115 such as AI model based on information or datawhich may be processed and sent from CBN 104. It should be noted thatthe machine learning makes predictions based on models generated andmaintained by various computational algorithms using historical data aswell as current data. A function of MLC 106 is that it is capable ofpushing information such as revised AI model or prediction model tovehicle 102 via a wireless communications network 114 constantly or inreal-time.

To identify or collect operator attention (or ability) of vehicle 102,an onboard AI model which could reside inside of VOC 123 receives atriggering event or events from built-in sensors such as ABS, wheelslippery, turning status, engine status, and the like. The triggeringevent or events may include, but not limited to, activation of ABS,rapid steering, rapid breaking, excessive wheel slip, activation ofemergency stop, and on. Upon receiving triggering events via vehicularstatus signals, the recording or recorded images captured by inwardfacing camera or 360 camera are forwarded to AIA system which resides atCBN 104.

In one embodiment, triggering events indicate an inattentional,distracted, and/or dangerous driver. For example, upon detecting apotential dangerous event, CBN 104 issues warning signal to driver oroperator 109 via, for instance, a haptic signal, or shock to operator109 notifying a potential collision. In addition, the dangerous event orevents are recorded for report. It should be noted that a reportdescribing driver's behavior as well as number occurrence relating todangerous events can be useful. For example, such report can be obtainedby insurance company for insurance auditing, by law enforcement foraccident prevention, by city engineers for traffic logistics, or bymedical stuff for patient safety.

During an operation, inward facing camera 126 captures facial images ofdriver or operator 109 including the location in which operator's eyesfocusing. Upon verifying with CBN 104, a focal direction 107 of operator109 is identified. After obtaining and processing external imagesrelating to focal direction 107, a possible trajectory 105 in which thelocation is looked at is obtained. Trajectory 105 and focal direction107 are subsequently processed and combined in accordance with storeddata in the cloud. The object, which is being looked at by operator 109,is identified. In this example, the object is a house 103 nearby theroad.

The AIA system records and examines various status such as pedalposition, steering wheel position, mirror setting, seat setting, engineRPM, whether the seat belts are clipped in, internal and externaltemperature, et cetera. With the advent of machine learning, a broadclass of derived data and metadata can be extracted from sensors and beused to improve the user experience of being in or driving a vehicle. Itshould be noted that the extracted data includes confidence andprobability metrics for each data element that the machine learningmodels generate. Such data, which changes in real-time, is presented toan application layer that can use the full context of vehicle inreal-time.

Operator 109, in one aspect, can be any driver capable of operating avehicle. For example, operator 109 can be a teen driver, elderly driver,professional race driver, fleet driver(s), and the like. The fleetdrivers can be, but not limited to, UPS (United Parcel Service) drivers,police officers, Federal Express drivers, taxi drivers, Uber drivers,Lyft drivers, delivery drivers, bus drivers, and the like.

An advantage of using an AIAS is to leverage cloud information as wellas embedded data to generate a report of VMP which can reduce trafficaccidents and enhance public safety by improving vehicle mechanicalcondition.

FIG. 1B is a block diagram 140 illustrating artificial intelligenceanalytic service (“AIAS”) capable of predicting vehicle mechanicalperformance (“VMP”) using a virtuous cycle in accordance with oneembodiment of the present invention. Diagram 140 illustrates a driver148, inward facing camera(s) 142, and exterior camera 144. In oneaspect, camera 142, also known as interior camera or 360 degree camera,monitors or captures driver's facial expression 146 and/or driver (oroperator) body language. Upon reading status 149 which indicates stablewith accelerometer, ahead with gaze, hands on steering wheel (notexting), the AIA model concludes that driver is behaving normally. Inone example, driver's identity (“ID”) can be verified using imagescaptured by interior camera 142.

AIA model, for example, is able to detect which direction driver 148 islooking, whether driver 148 is distracted, whether driver 148 istexting, whether identity of driver is determined via a facialrecognition process, and/or where driver 148 pays attention. It shouldbe noted that the car may contain multiple forward-facing cameras (or360-degree camera(s)) 144 capable of capturing a 360 degree view whichcan be used to correlate with other views to identify whether driver 148checks back-view mirror to see cars behind the vehicle or checks at sideview of vehicle when the car turns. Based on observed information, thelabeled data showing looking at the correct spots based on travelingroute of car can illustrate where the driver pays attention.Alternatively, the collected images or labeled data can be used toretrain the AIA model which may predict the safety rating for driver148.

During an operation, the interior images captured by inward facingcamera(s) 142 can show a location in which operator 148 is focusingbased on relative eye positions of operator 148. Once the direction oflocation such as direction 145 is identified, the AIA model obtainsexternal images captured by outward facing camera(s) 144. Afteridentifying image 145 is where operator pays attention based ondirection 145, the image 147 is recorded and process. Alternatively, ifAIA model expects operator 148 to look at the direction 145 based oncurrent speed and traffic condition while detecting operator 148actually looking at a house 141 based in trajectory view 143 based onthe captured images, a warning signal will be activated.

It should be noted that the labeled data should include various safetyparameters such as whether the driver looks left and/or right beforecrossing an intersection and/or whether the driver gazes at correctlocations while driving. The AIA model collects data from varioussensors, such as Lidar, radar, sonar, thermometers, audio detector,pressure sensor, airflow, optical sensor, infrared reader, speed sensor,altitude sensor, and the like, to establish operating environment. Theinformation can change based on occupant(s) behavior in the vehicle orcar. For example, if occupants are noisy, loud radio, shouting,drinking, eating, dancing, such behavior(s) can affect overallparameters as bad driving behavior.

FIG. 1C is a block diagram 130 illustrating a process of generatingreports relating to driver status using an AIA model via a virtuouscycle in accordance with one embodiment of the present invention.Diagram 130 includes vehicle 131, cloud 132, and subscriber 133 whereincloud 132 can further includes machine learning centers, historicaldriver data, and big data. In one embodiment, the AIA model, at leastpartially residing in cloud 132, is capable of providing AIAS tosubscriber(s) 133. It should be noted that the underlying concept of theexemplary embodiment(s) of the present invention would not change if oneor more blocks (circuit or elements) were added to or removed fromdiagram 130.

Vehicle 131 includes an infotainment unit 134, smart phone 135, VOC 136,and antenna 139. In one embodiment, infotainment unit 134 is coupled tohead-end unit such as VOC 136 to collect information about drivinghabit, skill, and/or ability associated with an operator based ondriver's condition, exterior environment, and internal equipment/vehiclestatus. Driver's condition includes driver ID, detected distractionssuch as talking over a phone, texting, occupant distraction, and thelike. Exterior environment refers to traffic condition, road condition,whether condition, and/or nearby drivers. The equipment or vehiclestatus indicates automobile mechanical conditions, such as ABSapplication, sharp steering, hard braking, sudden acceleration, tractioncontrol activation, windshield wipers movement, and/or airbagdeployment. The collected information is subsequently forwarded to cloud132 for processing.

Subscriber 133, in one example, can be insurance company, familymembers, law enforcement, car dealers, auto manufactures, and/or fleetcompanies such as Uber™ or UPS™. In one aspect, subscriber 133 is aninsurance company which wants to assess risks associated with certaingroup of drivers such as teen drivers or elderly drivers based onprediction reports generated by AIAS. For example, upon receipt ofcollected information from vehicle 131 to cloud 132 as indicated bynumeral 137, AIAS in cloud 132 generates a prediction report associatedwith a particular driver based on the driver's historical data as wellas big data. The prediction report is subsequently forwarded tosubscriber 133 from cloud 132 as indicated by number 138.

Smart phone 135, which can be an iPhone™ or Android™ phone, can be usedfor identifying driver's ID as well as provides communication to cloud132 via its cellular network access. Smart phone 135 can also be used tocouple to VOC 136 for facilitating hyperscale or data scale from clouddata to embedded data. Similarly, the embedded data can also be scaledbefore passing onto the cloud.

An advantage of employing AIAS is that it can provide a predictionrelating to VMP and/or VFP in connection to a group of vehicles toprovide vehicle intelligence to drivers as well as subscribers. Forexample, AIAS can predict or warn a possible mechanical failure or flattire in a near future.

FIG. 1D is a block diagram 150 illustrating an IDE 152 configured tohost various models for providing AIAS via a virtuous cycle inaccordance with one embodiment of the present invention. Diagram 150includes a core cloud system 151, IDE 152, driver data module 153,vehicle data module 154, geodata module 155, and applicationsmarketplace 156. While IDE 152 resides within core cloud system 151, IDE152 are configured to host various modules such as modules 153-156. Itshould be noted that the underlying concept of the exemplaryembodiment(s) of the present invention would not change if one or moreblocks (modules or elements) were added to or removed from diagram 150.

Driver data module 153, in one embodiment, includes a teen driverdetector, elder driver detector, and distracted driver detector. Theteen driver detector, in one example, monitors teen drivers based on aset of predefined rules. The predefined rules are often set by asubscriber such as an insurance company or parents. The elder driverdetector is responsible to monitor elderly drivers' ability to continuedriving according to a set of predefined rules. Based on the detectedand/or collected data, a prediction report can be automaticallygenerated and forwarded to subscriber(s) in an agreed or subscribed timeinterval. The distracted driver detector, in one embodiment, is used todetect distracted or disabled drivers and reports such distracteddrivers to authority for enhancing public safety. Upon collecting datafrom teen driver detector, elder driver detector, and distracted driverdetector, driver data module 153 forwards the collected data to IDE 152for AIAS processing.

Vehicle data module 154 includes a performance analyzer, predictivefailure analyzer, and fleet manager. The performance analyzer, in oneexample, is used to analyze and verify internal vehicle mechanicalperformance. For instance, tire slippage may be detected by theperformance analyzer. The predictive failure analyzer monitors vehiclemaintenance and/or repair before the failure of a particular part ordevice. The fleet manager, in one example, is used to monitor its fleetcars. For example, UPS tracks and/or Uber vehicles can be tracked andanalyzed to predict the operating efficiency and potential accidents.For example, after receipt of data from performance analyzer, predictivefailure analyzer, and fleet manager, vehicle data module 154 forwardsthe collected data to IDE 152 for AIAS processing.

Geodata module 155 includes a traffic router, hazard detector, andparking detector. The traffic router, in one aspect, is used to providea real-time alternative route in the present traffic congestion. In oneembodiment, the traffic router is able to communicate with other nearbyvehicles, stationary street cameras, and/or nearby drones to obtaincurrent situation. For instance, the traffic router can obtain reason(s)for congestion and based on the reason, such as an accident, roadconstruction, sinkhole, damaged bridge, or slow walker, an alternativeroute(s) may be provided. The Hazard detector, in one embodiment,detects hazard conditions such as potholes, chemical spills, and/or roadobstacles. The parking detector, in one embodiment, is able toautomatically identify where the vehicle can park, how long the vehiclehad parked, how much parking fee should be assessed. After receipt ofdata from traffic router, hazard detector, and parking detector, geodatamodule 155 forwards the collected data to IDE 152 for AIAS processing.

Applications marketplace 156 includes maintenance scheduler, microinsurance, and third-party modules. Applications marketplace 156, in oneaspect, facilitates third-party communications, software updates,applications, third-party modules, and the like. Third-party includesinsurance company, car deals, car repair shops, police, governmentagencies, city transportation, and/or other subscribers. In one aspect,Applications marketplace 156 is configured receive subscriptions as wellas sending prediction reports to subscribers based on a set ofpredefined time intervals.

In one embodiment, an AIA system capable of predicting an event or riskassociated with an operator driving a vehicle includes multiple interiorand exterior sensors, VOC, core cloud system or cloud. Cloud 151, in oneexample, includes an IDE 152 configured to host driver data module 153,vehicle data module 154, geodata module 155, and application marketplacemodule 156. In one aspect, the AIA system is able to predict a futureevent or potential risk based on current data, driver's historical data,and big data. The vehicle data indicates a collection of attributes,such as driving speed, braking frequency, sudden acceleration, ABStriggering, geographic locations, driver's personal records, and/ordetected distractions, to describe driver's behavior, skill, cognitivecondition, ability, and/or physical condition. The big data, in oneexample, refers to a set of data collected from large population havingsimilar attributes as the targeted driver's attributes. For example, atargeted driver is a teen age driver, the large population would be teenage drivers.

The interior and exterior sensors, in one example, installed on avehicle collect real-time data relating to external surroundings andinterior settings. The vehicle or car is operated by the driver ortargeted driver. The exterior sensors include outward facing cameras forcollecting external images representing a surrounding environment inwhich the vehicle operates. The interior sensors include inward facingcameras for collecting interior images inside of vehicle includingoperator facial expression as well as operator's attention. The externalimages include real-time images relating to road, buildings, trafficlights, pedestrian, or retailers. The interior images include a set ofinterior sensors obtaining data relating to at least one of operator'seyes, facial expression, driver, and passage. It should be noted thatinterior and exterior cameras can detect a direction in which theoperator is looking.

The VOC, in one example, is configured to generate a current datarepresenting current real-time status in accordance with the collecteddata. For instance, the VOC is able to identify operator's drivingability in response to the collected internal images and the collectedexternal images. In addition, driver or operator's ID can also beverified by the VOC.

Driver data module 153, in one aspect, includes a teen driver detector,elder driver detector, and distracted driver detector and is able toassess future predictions. The teen driver detector is able to reportteen's driving behavior to a subscriber based on the current data andhistorical data. For example, depending on the subscription supplied bythe subscriber, the report relating to teen's driving behavior orability can be periodically generated and sent to subscribers. The elderdriver detector is also able to report elder's driving behavior to asubscriber based on the current data and historical data.

Vehicle data module 154 contains a performance analyzer, predictivefailure analyzer, fleet manager for collecting vehicle related data.Geodata module 155 includes a traffic router, hazard detector, andparking detector for detecting locations or physical locations.Application marketplace module 156 contains a maintenance scheduler andmicro insurance for facilitating and/or enlisting subscribers. Forexample, the micro insurance is able to generate a report describingteen's driving behavior to an insurance subscriber according to the bigdata, current data, and historical data.

The AIA system, in one aspect, further includes audio sensors configuredto provide metadata relating to audio sound which occurs outside thevehicle or inside the vehicle. For example, the AIA system may includeexterior audio sensors collecting exterior sound outside of the vehicle.Similarly, multiple interior audio sensors may be used to collect soundinside of the vehicle. It should be noted that application marketplacemodule 156 includes a third-party module which is able to host variousthird-party applications, such as, but not limited to, interactiveadvertisement, driverless vehicle application, drone application, andthe like.

An advantage of using the AIA system is that it is able to facilitateAIAS to provide VMP using detected real-time data, historical data, andbig data.

FIG. 1E is a logic diagram 160 illustrating an AIA system configured toprovide a prediction relating to vehicle mechanical performance (“VMP”)via a virtuous cycle in accordance with one embodiment of the presentinvention. Diagram 160 includes a real-time information module 161,historical data 170, cloud big data 171, optimal module 173, predictionmodule 179, subscription module 181, and vehicle failure prediction(“VFP”) module 193. Optimal module 173, in one embodiment, includesoptimal module 176, race car module 178, and normal condition module180. It should be noted that the underlying concept of the exemplaryembodiment(s) of the present invention would not change if one or moreblocks (modules or elements) were added to or removed from diagram 160.

Real-time information module 161, in one aspect, includes mechanicaldata 162, external data 164, and internal data 166. Mechanical data 162is obtained by various sensors placed near or on the mechanical devices.For example, sensors may be placed around tire and/or wheel to obtainreal-time information relating to tire slippage, wheel slippage, unusualnoise, ABS (anti-skid braking system) deployment, and the like.Similarly, sensors can be placed around engine to read enginetemperature, vibration, sound, and so on. In one aspect, mechanical data162 includes readings or measurements of temperature, humidity,vibration, sound/noise, and the like.

External data 164, in one example, includes real-time informationrelating to read condition, weather condition, traffic congestions,nearby cars, traveling directions, and geographic locations of thevehicle. External data 164 are obtained and/or collected by exteriorsensors mounted on the vehicle. Internal or interior data 166 includesreal-time readings about interior of vehicle, such as, not limited to,driver, driver gaze, passengers, smoking, drinking, texting, talking,and/or reading phone(s). Interior data 166 are obtained and/or collectedby interior sensors and/or cameras. The component of current data 168,in one embodiment, gathers real-time inputs from mechanical data 162,external data 164, and internal data 166. The real-time inputs arecategorized, sorted, and organized whereby relevant current data isstored while irrelevant data is discarded.

Historical data 170, in one example, is the relevant data for thevehicle over a period of time. Historical data 170 can be stored in thecloud data. Alternatively, historical data 170 is stored locally. Also,a portion of historical data 170 is stored in the cloud while a portionof historical data 170 is embedded in the vehicle storage or VOC. Afunction of historical data 170 can provide a quick examination as towhether the current real-time data is normal or not in comparison withthe historical data.

Cloud big data 171 includes samples 174 and big data 172 wherein cloudbig data 171 is resided in the cloud or cloud computing. Big data 172,in aspect, include a large number of samples 174 that obtain from alarge pool of vehicles having similar attributes as the targetedvehicle. The attributes include, but not limited to, similar cars,similar total mileage, similar year produced, similar geographiclocation, similar features, similar mechanical functions, similarmaintenance records, similar drivers, et cetera. An advantage of usingthe big data is that it may contain information having similar problemsor failures for cars like the targeted vehicle. For example, big data172 may suggest to vehicle owner to take the vehicle to the dealerbecause many similar cars like the vehicle have overheating problems.

Optimal module 173, in one embodiment, includes an optimal condition176, race car condition 178, and normal condition 180. After receivingand evaluating inputs from current data 168, historical data 170, andbig data 172, optimal condition 176, based on a set of predefined rules,examines and determines whether the vehicle or targeted vehicle is inits optimal condition. In one example, the optimal condition means thevehicle meets performance requirements as manufactured and shows no signof wear and tear. If the optimal condition is met, the process proceedsto race car condition 178. If the optimal condition fails to meet, theprocess proceeds to normal condition 180 since certain level of wear andtear can be normal. Race car condition 178, in one aspect, providesinformation as to whether the vehicle is used for race or just normalordinary use. Normal condition 180 provides information as to whetherthe current condition is normal wear and tear, or whether the vehicle isabout to break down or fail.

Predication module 179 includes four (4) predictions 182-188 whereinprediction 1 receives signal from race car condition 178 indicating thevehicle is used for car race. Based on a set of predefined rules for carrace criteria, a prediction report is generated indicating various AIAindications. For example, AIA indications can show where and when thetire slipped while the vehicle was traveling at certain speed. Based onthe design of the vehicle and the road condition, the driver may havemade some mistakes. Such report is subsequently forwarded tosubscription module 181 indicating driver skills and/or mistakes asindicated by numeral 190.

Prediction 2 receives signals from race car condition 178 indicating thevehicle is not used for car race. Based on a set of predefined non-racecar rules, a prediction report is generated indicating various AIAindications which is subsequently sent to subscription module 181 suchas manufacture as a subscriber. For example, based on the report, themanufacture can better understand the quality of its vehicles as well asshortcomings.

Prediction 3 receives signal from normal condition 180 indicating thevehicle is normal even though some wears and tears are considerednormal. Based on a set of predefined rules for normal criteria, aprediction report is generated indicating various AIA indicationsindicating that certain wears and tears are normal and predicts the timeframe that the vehicle is likely to require maintenance. For example,AIA indications can show where and when the tire slipped while thevehicle was traveling at certain speed. Based on the design of thevehicle and the road condition, certain parts such as tire(s) mayrequire replacement at a predicted distance future. For example, theprediction reports may suggest to subscriber 192 that the tires need tobe replaced in three months.

Prediction 4 receives signal from normal condition 180 indicating thevehicle is not normal and certain mechanical parts may require services.Based on a set of predefined rules for VFP criteria, a prediction reportis generated indicating various AIA indications indicating that one ormore parts is about to fail. The process proceeds to VFP to handle thefailures.

Subscription module 181 includes driver 190, manufacture 191, and somesubscribers 192 which includes owner 194, repair shops 196, authority198, police department, bus companies, tracking companies, deliverycompanies, and so on. Authority 198 can be department of transportationwhich, for instance, wants to know the safety features of electricalcars. The prediction report can be helpful to DOT to assess whethercertain vehicle should be on the road or not.

An advantage of using VMP is that the vehicle is capable of initiatingfeedback to subscribers such as driver or manufactures or insurancecompany regarding the vehicle condition as well as vehicle safety usingits AIA system. For example, the feedback can help driver skill,manufactures, insurance rates, road safety, repair services, andautomatic maintenance scheduling.

FIG. 1F is a logic diagram 2000 illustrating an AIA system facilitatedconfigured to provide a vehicle failure prediction (“VFP”) via avirtuous cycle in accordance with one embodiment of the presentinvention. Diagram 2000 includes inputs 2002, predictive failureanalyzer 2193, prediction 2006, and various subscribers. Possiblefailure 193, which is the same or similar to VFP 193 shown in FIG. 1E,triggers a process of predictive failure analyzer 2193 since VMPindicates a possible failure(s). It should be noted that the underlyingconcept of the exemplary embodiment(s) of the present invention wouldnot change if one or more blocks (modules or elements) were added to orremoved from diagram 2000.

Input 2002 includes current data 168, historical data 170, and big data2172 wherein big data 2172 further connected to simples for similarvehicle 2008, samples for similar failure 2010, recalls for similarvehicle 2012, and samples for similar mileage 2014. It should be notedthat additional samples may be taken by big data 2172. Big data 2172, inaspect, include a large number of samples across a large geographic areacontaining a large pool of vehicles having similar attributes as thetargeted vehicle. The attributes include, but not limited to, similarcars, similar total mileage, similar year produced, similar geographiclocation, similar features, similar mechanical functions, similarmaintenance records, similar drivers, similar failures, similar recalls,similar reports, et cetera.

Based on the real-time data from current data 168, historical collectedinformation associated with the vehicle from historical data 170, andbig data 2172, predicative failure analyzer 2193, based on a set ofpredefined failure rules, generates predictions 2006. In one embodiment,predictive failure analyzer 2193 includes multiple AI models trained byvirtuous cycles. For example, a failure predictive AI model isconfigured to assessing a likelihood failure rate as well as failuretime based on current data, historical data, and big data.

Predications 2006 includes owner report 2020, manufacture report 2030,government agency report 2040, and repair shop report 2050. Forinstance, owner report 2020 is a summary status report for the vehicleowner(s) 2022 indicating the likelihood failure in near future. Thereport, in one embodiment, is configured to provide recommendations asrepairable versus trade-in assessment.

Manufacture report 2030 is a summary status report for the vehiclemanufacture(s) 2032 indicating when and where is the failure. Forexample, such status report can indicate a nationwide recall for fixingthe problem(s). Also, the failure predictive AI model may suggestimprovements for the new vehicles based on current data, historicaldata, and big data.

Government agency report 2040 is a summary status report for thegovernment agency 2042 such as department of motor vehicle indicatingwhether such vehicle should be allowed to drive on the public road. Forexample, a newly designed electrical vehicle may be fire hazards if itreaches certain speed. Assessing a license fee for such vehicle can bedifferent from other traditional vehicles. Also, the failure predictiveAI model may suggest improvements to the department of motor vehiclebefore a more reasonable licensing fee is assessed.

Repair shop report 2050 is a summary status report for the repairscheduling block 2052 indicating when and where the vehicle will like tofail and when the vehicle should have a repair performed to avoid thefailure. For example, when a dealer obtains such status report, itschedules an appointment with the driver or owner to repair the defectsor failure before the vehicle stops working. In one aspect, the failurepredictive AI model may automatically schedule an appointment with theowner to an available dealer or repair shop based on current data,historical data, and big data.

FIG. 1G is a logic diagram illustrating an AIA process 3000 facilitatedby AIA system configured to provide VMP using big data via a virtuouscycle in accordance with one embodiment of the present invention. Atblock 3160, the car or vehicle starts to move driving by a driver oroperator. After starting the vehicle, the VOC of vehicle, at block 3161,activates various sensors at block 3186, and retrieves local data atblock 3162. The local data, for example, includes last driver's ID,stored driver's fingerprint, and last known vehicle location before thevehicle stops. At block 3165, the driver ID is identified based on theinputs from local data at block 3162 and images detected by sensors atblock 3186. At block 3168, the AIA process is able to determine whetheran expected driver is verified. If the driver is the expected driver,the process proceeds to block 3171. If the driver is not the expecteddriver, the process proceeds to block 3169 for further identification.Upon identifying the driver, the driver's information is updated atblock 3170. For example, the driver is not the same driver before thecar or vehicle stops. It should be noted that the identification processcan use the AIA system supported by the virtuous cycle.

At block 3166, various interior and exterior images, audio sound, andinternal mechanical conditions are detected based on the input data fromlocal data at block 3162 and sensors at block 3186. At block 3167,location of the vehicle and moving direction of the vehicle can beidentified based on the geodata such as GPS (global positioning system)at block 3163, local data from block 3162, and images captured bysensors at block 3186. After collecting data from blocks 3166-3168, theprocess proceeds from block 3171 to block 3173. At block 3173, theprocess generates current data for VMP associated with the targetedvehicle according to collected data as well as a set of rules which canbe obtained from block 3172. The current data is subsequently uploadedat block 3174.

At block 3176, the AIA process performs AI analysis based on the currentdata from block 3174, historical data from block 3175, and big data fromblock 3164. Based on the AI analysis, the process provides a predictionat block 3178. Depending on the subscribers at block 3177, varioussubscribers 3180-3182 including manufacture 3180 will receive the reportrelating to the prediction. It should be noted that historical dataupdate module at block 3179 is used to update the historical data basedon the current data.

Note that the expected driver can be, but not limited to, companydriver, fleet driver, bus drivers, self-driving vehicles, drones, andthe like. It should be noted that the underlying concept of theexemplary logic diagram 3000 showing one embodiment(s) of the presentinvention would not change if one or more blocks (components orelements) were added to or removed from diagram 3000.

FIGS. 2A-2B are block diagrams 200 illustrating a virtuous cycle capableof facilitating AIAS using IA model in accordance with one embodiment ofthe present invention. Diagram 200, which is similar to diagram 100shown in FIG. 1A, includes a containerized sensor network 206,real-world scale data 202, and continuous machine learning 204. In oneembodiment, continuous machine learning 204 pushes real-time models tocontainerized sensor network 206 as indicated by numeral 210.Containerized sensor network 206 continuously feeds captured data orimages to real-world scale data 202 with uploading in real-time or in abatched format. Real-world scale data 202 provides labeled data tocontinuous machine learning 204 for constant model training as indicatedby numeral 212. It should be noted that the underlying concept of theexemplary embodiment(s) of the present invention would not change if oneor more blocks (or elements) were added to or removed from FIG. 2A.

The virtuous cycle illustrated in diagram 200, in one embodiment, isconfigured to implement IAS wherein containerized sensor network 206 issimilar to vehicle 102 as shown in FIG. 1A and real-world scale data 202is similar to CBN 104 shown in FIG. 1A. Also, continuous machinelearning 204 is similar to MCL 106 shown in FIG. 1A. In one aspect,containerized sensor network 206 such as an automobile or car contains acontainerized sensing device capable of collecting surroundinginformation or images using onboard sensors or sensor network when thecar is in motion. Based on the IA model, selective recording thecollected surrounding information is selectively recorded to a localstorage or memory.

Real-world scale data 202, such as cloud or CBN, which is wirelesslycoupled to the containerized sensing device, is able to correlate withcloud data and recently obtained IA data for producing labeled data. Forexample, real-world scale data 202 generates IA labeled data based onhistorical IA cloud data and the surrounding information sent from thecontainerized sensing device.

Continuous machine learning 204, such as MLC or cloud, is configured totrain and improve IA model based on the labeled data from real-worldscale data 202. With continuous gathering data and training IA model(s),the IAS will be able to learn, obtain, and/or collect all available IAsfor the population samples.

In one embodiment, a virtuous cycle includes partition-able MachineLearning networks, training partitioned networks, partitioning a networkusing sub-modules, and composing partitioned networks. For example, avirtuous cycle involves data gathering from a device, creatingintelligent behaviors from the data, and deploying the intelligence. Inone example, partition idea includes knowing the age of a driver whichcould place or partition “dangerous driving” into multiple models andselectively deployed by an “age detector.” An advantage of using suchpartitioned models is that models should be able to perform a better jobof recognition with the same resources because the domain of discourseis now smaller. Note that, even if some behaviors overlap by age, thepartitioned models can have common recognition components.

It should be noted that more context information collected, a better jobof recognition can be generated. For example, “dangerous driving” can befurther partitioned by weather condition, time of day, trafficconditions, et cetera. In the “dangerous driving” scenario, categoriesof dangerous driving can be partitioned into “inattention”, “aggressivedriving”, “following too closely”, “swerving”, “driving too slowly”,“frequent breaking”, deceleration, ABS event, et cetera.

For example, by resisting a steering behavior that is erratic, the cargives the driver direct feedback on their behavior - if the resistanceis modest enough then if the steering behavior is intentional (such astrying to avoid running over a small animal) then the driver is stillable to perform their irregular action. However, if the driver istexting or inebriated then the correction may alert them to theirbehavior and get their attention. Similarly, someone engaged in “roadrage” who is driving too close to another car may feel resistance on thegas pedal. A benefit of using IAS is to identify consequences of adriver's “dangerous behavior” as opposed to recognizing the causes(texting, etc.). The Machine Intelligence should recognize the causes aspart of the analysis for offering corrective action.

In one aspect, a model such as IA model includes some individual blocksthat are trained in isolation to the larger problem (e.g. weatherdetection, traffic detection, road type, etc.). Combining the blocks canproduce a larger model. Note that the sample data may include behaviorsthat are clearly bad (ABS event, rapid deceleration, midline crossing,being too close to the car in front, etc.). In one embodiment, one ormore sub-modules are built. The models include weather conditiondetection and traffic detection for additional modules intelligence,such as “correction vectors” for “dangerous driving.”

An advantage of using a virtuous cycle is that it can learn and detectobject such as IA in the real world.

FIG. 2B is a block diagram 230 illustrating an alternative exemplaryvirtuous cycle capable of detecting IA in accordance with one embodimentof the present invention. Diagram 230 includes external data source 234,sensors 238, crowdsourcing 233, and intelligent model 239. In oneaspect, components/activities above dotted line 231 are operated incloud 232, also known as in-cloud component. Components/activities belowdotted line 231 are operated in car 236, also known as in-device orin-car component. It should be noted that the underlying concept of theexemplary embodiment(s) of the present invention would not change if oneor more blocks (or elements) were added to or removed from FIG. 2B.

In one aspect, in-cloud components and in-device components coordinateto perform desirable user specific tasks. While in-cloud componentleverages massive scale to process incoming device information, cloudapplications leverage crowd sourced data to produce applications.External data sources can be used to contextualize the applications tofacilitate intellectual crowdsourcing. For example, in-car (or in-phoneor in-device) portion of the virtuous cycle pushes intelligent datagathering to the edge application. In one example, edge applications canperform intelligent data gathering as well as intelligent in-carprocessing. It should be noted that the amount of data gathering mayrely on sensor data as well as intelligent models which can be loaded tothe edge.

FIG. 3 is a block diagram illustrating a cloud based network usingcrowdsourcing approach to improve IA model(s) for AIAS in accordancewith one embodiment of the present invention. Diagram 300 includespopulation of vehicles 302, sample population 304, model deployment 306,correlation component 308, and cloud application 312. It should be notedthat the underlying concept of the exemplary embodiment(s) of thepresent invention would not change if one or more blocks (or samples)were added to or removed from FIG. 3.

Crowdsourcing is a process of using various sourcing or specific modelsgenerated or contributed from other cloud or Internet users forachieving needed services. For example, crowdsourcing relies on theavailability of a large population of vehicles, phones, or other devicesto source data 302. For example, a subset of available devices such assample 304 is chosen by some criterion such as location to perform datagathering tasks. To gather data more efficiently, intelligent models aredeployed to a limited number of vehicles 306 for reducing the need oflarge uploading and processing a great deal of data in the cloud. Itshould be noted that the chosen devices such as cars 306 monitor theenvironment with the intelligent model and create succinct data aboutwhat has been observed. The data generated by the intelligent models isuploaded to the correlated data store as indicated by numeral 308. Itshould be noted that the uploading can be performed in real-time forcertain information or at a later time for other types of informationdepending on the need as well as condition of network traffic.

Correlated component 308 includes correlated data storage capable ofproviding a mechanism for storing and querying uploaded data. Cloudapplications 312, in one embodiment, leverage the correlated data toproduce new intelligent models, create crowd sourced applications, andother types of analysis.

FIG. 4 is a block diagram 400 illustrating an IA model or AIA systemusing the virtuous cycle in accordance with one embodiment of thepresent invention. Diagram 400 includes a correlated data store 402,machine learning framework 404, and sensor network 406. Correlated datastore 402, machine learning framework 404, and sensor network 406 arecoupled by connections 410-416 to form a virtuous cycle as indicated bynumeral 420. It should be noted that the underlying concept of theexemplary embodiment(s) of the present invention would not change if oneor more blocks (circuit or elements) were added to or removed from FIG.4.

In one embodiment, correlated data store 402 manages real-time streamsof data in such a way that correlations between the data are preserved.Sensor network 406 represents the collection of vehicles, phones,stationary sensors, and other devices, and is capable of uploadingreal-time events into correlated data store 402 via a wirelesscommunication network 412 in real-time or in a batched format. In oneaspect, stationary sensors include, but not limited to, municipalcameras, webcams in offices and buildings, parking lot cameras, securitycameras, and traffic cams capable of collecting real-time images.

The stationary cameras such as municipal cameras and webcams in officesare usually configured to point to streets, buildings, parking lotswherein the images captured by such stationary cameras can be used foraccurate labeling. To fuse between motion images captured by vehiclesand still images captured by stationary cameras can track object(s) suchas car(s) more accurately. Combining or fusing stationary sensors andvehicle sensors can provide both labeling data and historical stationarysampling data also known as stationary “fabric”. It should be noted thatduring the crowdsourcing applications, fusing stationary data (e.g.stationary cameras can collect vehicle speed and position) withreal-time moving images can improve ML process.

Machine Learning (“ML”) framework 404 manages sensor network 406 andprovides mechanisms for analysis and training of ML models. ML framework404 draws data from correlated data store 402 via a communicationnetwork 410 for the purpose of training modes and/or labeled dataanalysis. ML framework 404 can deploy data gathering modules to gatherspecific data as well as deploy ML models based on the previouslygathered data. The data upload, training, and model deployment cycle canbe continuous to enable continuous improvement of models.

FIG. 5 is a block diagram 500 illustrating an exemplary process ofcorrelating data for AIAS in accordance with one embodiment of thepresent invention. Diagram 500 includes source input 504, real-time datamanagement 508, history store 510, and crowd sourced applications512-516. In one example, source input 504 includes cars, phones,tablets, watches, computers, and the like capable of collecting massiveamount of data or images which will be passed onto real-time datamanagement 508 as indicated by numeral 506. It should be noted that theunderlying concept of the exemplary embodiment(s) of the presentinvention would not change if one or more blocks (or elements) wereadded to or removed from FIG. 5.

In one aspect, a correlated system includes a real-time portion and abatch/historical portion. The real-time part aims to leverage new datain near or approximately real-time. Real-time component or management508 is configured to manage a massive amount of influx data 506 comingfrom cars, phones, and other devices 504. In one aspect, after ingestingdata in real-time, real-time data management 508 transmits processeddata in bulk to the batch/historical store 510 as well as routes thedata to crowd sourced applications 512-516 in real-time.

Crowd sourced applications 512-516, in one embodiment, leveragereal-time events to track, analyze, and store information that can beoffered to user, clients, and/or subscribers. Batch-Historical side ofcorrelated data store 510 maintains a historical record of potentiallyall events consumed by the real-time framework. In one example,historical data can be gathered from the real-time stream and it can bestored in a history store 510 that provides high performance, low cost,and durable storage. In one aspect, real-time data management 508 andhistory store 510 coupled by a connection 502 are configured to performIA data correlation as indicated by dotted line.

FIG. 6 is a block diagram illustrating an exemplary process of real-timedata management for AI model used for AIAS in accordance with oneembodiment of the present invention. Diagram 600 includes data input602, gateway 606, normalizer 608, queue 610, dispatcher 616, storageconversion 620, and historical data storage 624. The process ofreal-time data management further includes a component 614 for publishand subscribe. It should be noted that the underlying concept of theexemplary embodiment(s) of the present invention would not change if oneor more blocks (circuit or elements) were added to or removed from FIG.6.

The real-time data management, in one embodiment, is able to handlelarge numbers (i.e., 10's of millions) of report events to the cloud asindicated by numeral 604. API (application program interface) gateway606 can handle multiple functions such as client authentication and loadbalancing of events pushed into the cloud. The real-time data managementcan leverage standard HTTP protocols. The events are routed to statelessservers for performing data scrubbing and normalization as indicated bynumeral 608. The events from multiple sources 602 are aggregatedtogether into a scalable/durable/consistent queue as indicated bynumeral 610. An event dispatcher 616 provides a publish/subscribe modelfor crowd source applications 618 which enables each application to lookat a small subset of the event types. The heterogeneous event stream,for example, is captured and converted to files for long-term storage asindicated by numeral 620. Long-term storage 624 provides a scalable anddurable repository for historical data.

FIG. 7 is a block diagram 700 illustrating a crowd sourced applicationmodel for AI model for AIAS in accordance with one embodiment of thepresent invention. Diagram 700 includes a gateway 702, event handler704, state cache 706, state store 708, client request handler 710,gateway 712, and source input 714. In one example, gateway 702 receivesan event stream from an event dispatcher and API gateway 712 receivesinformation/data from input source 714. It should be noted that theunderlying concept of the exemplary embodiment(s) of the presentinvention would not change if one or more blocks (or elements) wereadded to or removed from FIG. 7.

The crowd sourced application model, in one embodiment, facilitatesevents to be routed to a crowd source application from a real-time datamanager. In one example, the events enter gateway 702 using a simplepush call. Note that multiple events are handled by one or more servers.The events, in one aspect, are converted into inserts or modificationsto a common state store. State store 708 is able to hold data frommultiple applications and is scalable and durable. For example, Statestore 708, besides historical data, is configured to store present data,information about “future data”, and/or data that can be shared acrossapplications such as predictive AI (artificial intelligence).

State cache 706, in one example, is used to provide fast access tocommonly requested data stored in state store 708. Note that applicationcan be used by clients. API gateway 712 provides authentication and loadbalancing. Client request handler 710 leverages state store 708 forproviding client data.

In an exemplary embodiment, an onboard IA model is able to handlereal-time IA detection based on triggering events. For example, after MLmodels or IA models for IA detection have been deployed to all or mostof the vehicles, the deployed ML models will report to collected dataindicating IAS for facilitating issuance of real-time warning fordangerous event(s). The information or data relating to the real-timedangerous event(s) or IAS is stored in state store 708. Vehicles 714looking for IA detection can, for example, access the IAS using gateway712.

FIG. 8 is a block diagram 800 illustrating a method of storing AIrelated data using a geo-spatial objective storage for AIAS inaccordance with one embodiment of the present invention. Diagram 800includes gateway 802, initial object 804, put call 806, find call 808,get call 810, SQL (Structured Query Language) 812, non-SQL 814, andgeo-spatial object storage 820. It should be noted that the underlyingconcept of the exemplary embodiment(s) of the present invention wouldnot change if one or more blocks (circuit or elements) were added to orremoved from FIG. 8.

Geo-spatial object storage 820, in one aspect, stores or holds objectswhich may include time period, spatial extent, ancillary information,and optional linked file. In one embodiment, geo-spatial object storage820 includes UUID (universally unique identifier) 822, version 824,start and end time 826, bounding 828, properties 830, data 832, andfile-path 834. For example, while UUID 822 identifies an object, allobjects have version(s) 824 that allow schema to change in the future.Start and end time 826 indicates an optional time period with a starttime and an end time. An optional bounding geometry 828 is used tospecify spatial extent of an object. An optional set of properties 830is used to specify name-value pairs. Data 832 can be binary data. Anoptional file path 834 may be used to associate with the object of afile containing relevant information such as MPEG (Moving PictureExperts Group) stream.

In one embodiment, API gateway 802 is used to provide access to theservice. Before an object can be added to the store, the object isassigned an UUID which is provided by the initial object call. Once UUIDis established for a new object, the put call 804 stores the objectstate. The state is stored durably in Non-SQL store 814 along with UUID.A portion of UUID is used as hash partition for scale-out. The indexableproperties includes version, time duration, bounding, and propertieswhich are inserted in a scalable SQL store 812 for indexing. The Non-SQLstore 814 is used to contain the full object state. Non-SQL store 814 isscaled-out using UUID as, for example, a partition key.

SQL store 812 is used to create index tables that can be used to performqueries. SQL store 812 may include three tables 816 containinginformation, bounding, and properties. For example, information holds aprimary key, objects void, creation timestamp, state of object andobject properties “version” and “time duration.” Bounding holds thebounding geometry from the object and the id of the associatedinformation table entry. Properties hold property name/value pairs fromthe object stored as one name/value pair per row along with ID ofassociated info table entry.

Find call 808, in one embodiment, accepts a query and returns a resultset, and issues a SQL query to SQL store 812 and returns a result setcontaining UUID that matches the query.

FIG. 9 is a block diagram 900 illustrating an exemplary approach ofanalysis engine analyzing collected data for AIAS in accordance with oneembodiment of the present invention. Diagram 900 includes history store902, analysis engine 904, and geo-spatial object store 906. It should benoted that the underlying concept of the exemplary embodiment(s) of thepresent invention would not change if one or more blocks (circuit orelements) were added to or removed from FIG. 9.

In one aspect, diagram 900 illustrates analysis engine 904 containing MLtraining component capable of analyzing labeled data based on real-timecaptured IA data and historical data. The data transformation engine, inone example, interacts with Geo-spatial object store 906 to locaterelevant data and with history store to process the data. Optimally, thetransformed data may be stored.

It should be noted that virtuous cycle employing ML training componentto provide continuous model training using real-time data as well ashistorical samples, and deliver IA detection model for one or moresubscribers. A feature of virtuous cycle is able to continuous traininga model and able to provide a real-time or near real-time result. Itshould be noted that the virtuous cycle is applicable to various otherfields, such as, but not limited to, business intelligence, lawenforcement, medical services, military applications, and the like.

FIG. 10 is a block diagram 1000 illustrating an exemplary containerizedsensor network used for sensing information for AIAS in accordance withone embodiment of the present invention. Diagram 1000 includes a sensorbus 1002, streaming pipeline 1004, and application layer 1006 whereinsensor bus 1002 is able to receive low-bandwidth sources andhigh-bandwidth sources. Streaming pipeline 1004, in one embodiment,includes ML capable of generating unique model such as model 1008. Itshould be noted that the underlying concept of the exemplaryembodiment(s) of the present invention would not change if one or moreblocks (circuit or elements) were added to or removed from FIG. 10.

FIG. 11 is a block diagram 1100 illustrating a processing device, VOC,and/or computer(s) which can be installed in a vehicle to supportonboard cameras, CAN (Controller Area Network) bus, Inertial MeasurementUnits, Lidar, et cetera for facilitating virtuous cycle in accordancewith one embodiment of the present invention. Computer system or IAS1100 can include a processing unit 1101, an interface bus 1112, and aninput/output (“IO”) unit 1120. Processing unit 1101 includes a processor1102, a main memory 1104, a system bus 1111, a static memory device1106, a bus control unit 1105, I/O element 1130, and IAS element 1185.It should be noted that the underlying concept of the exemplaryembodiment(s) of the present invention would not change if one or moreblocks (circuit or elements) were added to or removed from FIG. 11.

Bus 1111 is used to transmit information between various components andprocessor 1102 for data processing. Processor 1102 may be any of a widevariety of general-purpose processors, embedded processors, ormicroprocessors such as ARM® embedded processors, Intel® Core™ Duo,Core™ Quad, Xeon®, Pentium microprocessor, Motorola™ 68040, AMD® familyprocessors, or Power PC™ microprocessor.

Main memory 1104, which may include multiple levels of cache memories,stores frequently used data and instructions. Main memory 1104 may beRAM (random access memory), MRAM (magnetic RAM), or flash memory. Staticmemory 1106 may be a ROM (read-only memory), which is coupled to bus1111, for storing static information and/or instructions. Bus controlunit 1105 is coupled to buses 1111-1112 and controls which component,such as main memory 1104 or processor 1102, can use the bus. Bus controlunit 1105 manages the communications between bus 1111 and bus 1112.

I/O unit 1120, in one embodiment, includes a display 1121, keyboard1122, cursor control device 1123, and communication device 1125. Displaydevice 1121 may be a liquid crystal device, cathode ray tube (“CRT”),touch-screen display, or other suitable display device. Display 1121projects or displays images of a graphical planning board. Keyboard 1122may be a conventional alphanumeric input device for communicatinginformation between computer system 1100 and computer operator(s).Another type of user input device is cursor control device 1123, such asa conventional mouse, touch mouse, trackball, or other type of cursorfor communicating information between system 1100 and user(s).

IA element 1185, in one embodiment, is coupled to bus 1111, andconfigured to interface with the virtuous cycle for facilitating IAdetection(s). For example, if system 1100 is installed in a car, IAelement 1185 is used to operate the IA model as well as interface withthe cloud based network. If system 1100 is placed at the cloud basednetwork, IA element 1185 can be configured to handle the correlatingprocess for generating labeled data. Communication device 1125 iscoupled to bus 1111 for accessing information from remote computers orservers, such as server 104 or other computers, through wide-areanetwork 102. Communication device 1125 may include a modem or a networkinterface device, or other similar devices that facilitate communicationbetween computer 1100 and the network. Computer system 1100 may becoupled to a number of servers via a network infrastructure such as theInternet.

The exemplary embodiment of the present invention includes variousprocessing steps, which will be described below. The steps of theembodiment may be embodied in machine or computer executableinstructions. The instructions can be used to cause a general purpose orspecial purpose system, which is programmed with the instructions, toperform the steps of the exemplary embodiment of the present invention.Alternatively, the steps of the exemplary embodiment of the presentinvention may be performed by specific hardware components that containhard-wired logic for performing the steps, or by any combination ofprogrammed computer components and custom hardware components.

FIG. 12 is a flowchart 1200 illustrating a process of AIAS for providinga report of VMP predicting vehicle status in accordance with oneembodiment of the present invention. At block 1202, a process able topredict an event relating to machinal performance using data obtainedfrom interior and exterior sensors, VOC, and cloud data activatesinterior and exterior sensors mounted on a vehicle operated by a driverfor obtaining current data relating to external surroundings, interiorsettings, and internal mechanical conditions of the vehicle. Forexample, after enabling a set of outward facing cameras mounted on thevehicle for recording external surrounding images representing ageographic environment, one or more inward facing cameras mounted in thevehicle is initiated for collecting interior images of the vehicle.Also, a set of internal sensors attached to various mechanicalcomponents is activated for measuring temperatures, functionalities, oraudio sounds associated with mechanical components within the vehicle.The process, in one embodiment, is capable of detecting driver'sresponse time based on a set of identified road conditions andinformation from a controller area network (“CAN”) bus of the vehicle.The real-time data relating to vehicle performance, road condition,traffic congestion, and weather condition is recorded.

At block 1204, the current data is forwarded to VOC for generating acurrent vehicle status representing substantially real-time vehicleperformance in accordance with the current data.

At block 1206, a historical data associated with the vehicle includingmechanical condition is retrieved. Note that the historical data isupdated in response to the current data.

At block 1208, a normal condition signal is issued when the currentvehicle status does not satisfy with optimal condition based on thehistorical data. In one aspect, after uploading the current vehiclestatus to a vehicle performance predictor which resides at leastpartially at a cloud via a communications network, the big data isobtained from the cloud wherein the big data represents large carsamples having similar attributes as the vehicle. For example, the bigdata accumulates information from cars with similar brands, similarmileages, similar years, similar geographic location, and similardrivers. The current vehicle status is compared with the big data andthe historical data to assess whether the vehicle operates in a normalcondition.

At block 1210, a race car condition is issued when the current vehiclestatus meets with the optimal condition based on the historical data. Inone example, the current vehicle status is forwarded to a subscriber forevaluating driver's driving skill. The current vehicle status can alsobe forwarded to a subscriber for assessing normal wearing and tearing.Alternatively, a subscriber schedules a maintenance or repairappointment with the driver based on the current vehicle status. In oneembodiment, a manufacture can initiate a recall for automobiles similarto the vehicle at least partially based on the current vehicle status.

While particular embodiments of the present invention have been shownand described, it will be obvious to those of ordinary skills in the artthat based upon the teachings herein, changes and modifications may bemade without departing from this exemplary embodiment(s) of the presentinvention and its broader aspects. Therefore, the appended claims areintended to encompass within their scope all such changes andmodifications as are within the true spirit and scope of this exemplaryembodiment(s) of the present invention.

What is claimed is:
 1. A method, comprising: activating a plurality ofsensors associated with a vehicle; obtaining data from the plurality ofsensors relating to external surroundings, interior settings, orinternal mechanical conditions of the vehicle; generating a currentvehicle status representing current vehicle performance based on theobtained data; retrieving historical data associated with the vehicle;and issuing a normal condition signal in response to the current vehiclestatus failing to satisfy an optimal condition based on the historicaldata.
 2. The method of claim 1, further comprising: issuing a race-carcondition signal indicating the vehicle is used for racing in responseto the current vehicle status satisfying the optimal condition based onthe historical data.
 3. The method of claim 1, further comprising:sending a performance report to a third party indicating a currentmechanical condition of the vehicle based on the current vehicleperformance and the historical data.
 4. The method of claim 1, whereinactivating the plurality of sensors includes: enabling one or moreoutward-facing cameras on the vehicle to record an environment outsidethe vehicle.
 5. The method of claim 1, wherein activating the pluralityof sensors includes: enabling one or more inward-facing cameras on thevehicle to record an environment inside the vehicle.
 6. The method ofclaim 1, wherein activating the plurality of sensors includes: recordingreal-time data relating to at least one of vehicle performance, roadconditions, traffic congestion, or weather conditions. 7 The method ofclaim 1, wherein issuing the normal condition signal includes:determining that the current vehicle status fails to satisfy the optimalcondition based on the current vehicle status failing meet performancerequirements as manufactured based on the historical data.
 8. The methodof claim 1, wherein issuing the normal condition signal includes:determining that the current vehicle status fails to satisfy the optimalcondition based on the current vehicle status indicating signs of wearand tear.
 9. The method of claim 1, further comprising: scheduling amaintenance appointment for the vehicle based on the current vehiclestatus.
 10. The method of claim 1, further comprising: initiating amanufacture recall related to the vehicle at least partially based onthe current vehicle status.
 11. A computing system, comprising: a memorythat stores computer instructions; and a processor that executes thecomputer instruction to: activate a plurality of sensors associated witha vehicle; obtain data from the plurality of sensors relating toexternal surroundings, interior settings, or internal mechanicalconditions of the vehicle; generate a current vehicle statusrepresenting current vehicle performance based on the obtained data;retrieve historical data associated with the vehicle; and issue arace-car condition signal indicating the vehicle is used for racing inresponse to the current vehicle status satisfying an optimal conditionbased on the historical data.
 12. The computing system of claim 11,wherein the processor executes further computer instructions to: issue anormal condition signal in response to the current vehicle statusfailing to satisfy an optimal condition based on the historical data.13. The computing system of claim 11, wherein the processor executesfurther computer instructions to: send a performance report to a thirdparty indicating a current mechanical condition of the vehicle based onthe current vehicle performance and the historical data.
 14. Thecomputing system of claim 11, wherein the processor executes furthercomputer instructions to: send a performance report to a third partyindicating skills or mistakes associated with a driver of the vehiclebased on the current vehicle performance and the historical data. 15.The computing system of claim 11, wherein the processor issues therace-car condition signal by further executing further computerinstructions to: determine that the current vehicle status satisfies theoptimal condition based on the current vehicle status meetingperformance requirements as manufactured and failing to indicate signsof wear and tear.
 16. A non-transitory computer-readable storage mediumhaving stored thereon instructions that, when executed by a processor,cause the processor to perform actions, the actions comprising:activating a plurality of sensors associated with a vehicle; obtainingdata from the plurality of sensors relating to external surroundings,interior settings, or internal mechanical conditions of the vehicle;generating a current vehicle status representing current vehicleperformance based on the obtained data; retrieving historical dataassociated with the vehicle; and sending a performance report to a thirdparty indicating a current mechanical condition of the vehicle based onthe current vehicle performance and the historical data.
 17. Thenon-transitory computer-readable storage medium of claim 16, furthercomprising: issuing a race-car condition signal indicating the vehicleis used for racing in response to the current vehicle status satisfyingthe optimal condition based on the historical data.
 18. Thenon-transitory computer-readable storage medium of claim 16, furthercomprising: issuing a normal condition signal in response to the currentvehicle status failing to satisfy an optimal condition based on thehistorical data.
 19. The non-transitory computer-readable storage mediumof claim 16, further comprising: scheduling a maintenance appointmentfor the vehicle based on the current vehicle status.
 20. Thenon-transitory computer-readable storage medium of claim 16, furthercomprising: initiating a manufacture recall related to the vehicle atleast partially based on the current vehicle status.